A CMOS approach to compressed-domain image acquisition

A hardware implementation of a real-time compressed-domain image acquisition system is demonstrated. The system performs front-end computational imaging, whereby the inner product between an image and an arbitrarily-specified mask is implemented in silicon. The acquisition system is based on an intelligent readout integrated circuit (iROIC) that is capable of providing independent bias voltages to individual detectors, which enables implementation of spatial multiplication with any prescribed mask through a bias-controlled response-modulation mechanism. The modulated pixels are summed up in the image grabber to generate the compressed samples, namely aperturecoded coefficients, of an image. A rigorous bias-selection algorithm is presented to the readout circuit, which exploits the bias-dependent nature of the imager’s responsivity. Proven functionality of the hardware in transform coding compressed image acquisition, silicon-level compressive sampling, in pixel nonuniformity correction and hardware-level implementation of region-based enhancement is demonstrated. © 2017 Optical Society of America OCIS codes: (110.0110) Imaging systems; (100.2000) Digital image processing; (280.4788) Optical sensing and sensors; (040.0040) Detectors. References and links 1. M. Prastawa, E. Bullitt, S. Ho, and G. Gerig, “A brain tumor segmentation framework based on outlier detection,” Medical image analysis 8, 275–283 (2004). 2. M. P. Edgar, G. M. Gibson, R. W. Bowman, B. Sun, N. Radwell, K. J. Mitchell, S. S. Welsh, and M. J. Padgett, “Simultaneous real-time visible and infrared video with single-pixel detectors,” Scientific reports 5 (2015). 3. M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressed sensing MRI,” IEEE Sig. Proc. Mag. 25, 72–82 (2008). 4. F. Shao, W. Lin, G. Jiang, and Q. Dai, “Models of monocular and binocular visual perception in quality assessment of stereoscopic images,” IEEE T. Comput. Imag. 2, 123–135 (2016). 5. S. V. Venkatakrishnan, L. F. Drummy, M. Jackson, M. De Graef, J. Simmons, and C. A. Bouman, “Model-based iterative reconstruction for bright-field electron tomography,” IEEE T. Comput. Imag. 1, 1–15 (2015). 6. A. Chowdhury, R. Darveaux, J. Tome, R. Schoonejongen, M. Reifel, A. De Guzman, S. S. Park, Y. W. Kim, and H. W. Kim, “Challenges of megapixel camera module assembly and test,” in “Proceedings Electronic Components and Technology, 2005. ECTC’05.”, (IEEE, 2005), pp. 1390–1401. 7. N. Nakano, R. Nishimura, H. Sai, A. Nishizawa, and H. Komatsu, “Digital still camera system for megapixel CCD,” IEEE T. Consum. Electron. 44, 581–586 (1998). 8. C. F. Weiman and J. M. Evans Jr, “Digital image compression employing a resolution gradient,” (1992). US Patent 5,103,306. 9. P. T. Barrett, “Method for image compression on a personal computer,” (1994). US Patent 5,287,420. 10. J. G. Daugman, “High confidence visual recognition of persons by a test of statistical independence,” IEEE T. Pattern Anal. 15, 1148–1161 (1993). 11. A. Gandomi and M. Haider, “Beyond the hype: Big data concepts, methods, and analytics,” Int. J. Inform. Manage. 35, 137–144 (2015). 12. Y. Oike, M. Ikeda, and K. Asada, “Design and implementation of real-time 3-D image sensor with 640× 480 pixel resolution,” IEEE J. Solid-St. Circ. 39, 622–628 (2004). 13. R. LiKamWa, B. Priyantha, M. Philipose, L. Zhong, and P. Bahl, “Energy characterization and optimization of image sensing toward continuous mobile vision,” in “Proceeding of the 11th annual international conference on Mobile systems, applications, and services,” (ACM, 2013), pp. 69–82. 14. I. Cevik, X. Huang, H. Yu, M. Yan, and S. U. Ay, “An ultra-low power CMOS image sensor with on-chip energy harvesting and power management capability,” Sensors 15, 5531–5554 (2015). 15. M. Dadkhah, M. J. Deen, and S. Shirani, “Compressive sensing image sensors-hardware implementation,” Sensors 13, 4961–4978 (2013). 16. R. G. Baraniuk, “Compressive sensing,” IEEE Sig. Proc. Mag. 24 (2007). 17. J. Ribas-Corbera and S. Lei, “Rate control in DCT video coding for low-delay communications,” IEEE T. Circuits Syst. Video Technol. 9, 172–185 (1999). 18. M. Leinonen, M. Codreanu, and M. Juntti, “Compressed acquisition and progressive reconstruction of multi-dimensional correlated data in wireless sensor networks,” in “2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),” (IEEE, 2014), pp. 6449–6453. 19. G. R. C. Fiorante, P. Zarkesh-Ha, J. Ghasemi, and S. Krishna, “Spatio-temporal tunable pixels for multi-spectral infrared imagers,” in “2013 IEEE 56th International Midwest Symposium on Circuits and Systems (MWSCAS),” (IEEE, 2013), pp. 317–320. 20. M. Bhattarai, J. Ghasemi, G. R. Fiorante, P. Zarkesh-Ha, S. Krishna, and M. M. Hayat, “Intelligent bias-selection method for computational imaging on a CMOS imager,” in “2016 IEEE Photonics Conference,” (2016). 21. J. Lee, S. Lim, and G. Han, “A 10b column-wise two-step single-slope adc for high-speed cmos image sensor,” in “Proc. IEEE Int. Image sensor Workshop, Ogunquit, ME,” (Citeseer, 2007), pp. 196–199. 22. M. Lustig, D. Donoho, and J. M. Pauly, “Sparse MRI: The application of compressed sensing for rapid MR imaging,” Magnetic resonance in medicine 58, 1182–1195 (2007). 23. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. E. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Sig. Proc. Mag. 25, 83 (2008). 24. J. B. Sampsell, “Digital micromirror device and its application to projection displays,” J. Vac. Sci. Technol. B 12, 3242–3246 (1994). 25. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Optics express 21, 10526–10545 (2013). 26. Y. Oike and A. El Gamal, “A 256× 256 CMOS image sensor with ∆Σ-based single-shot compressed sensing,” in “2012 IEEE International Solid-State Circuits Conference,” (IEEE, 2012), pp. 386–388. 27. H. Tian, “Noise analysis in CMOS image sensors,” Ph.D. thesis, Citeseer (2000). 28. M. Bigas, E. Cabruja, J. Forest, and J. Salvi, “Review of CMOS image sensors,” Microelectron. J. 37, 433–451 (2006). 29. S. Mendis, S. E. Kemeny, and E. R. Fossum, “CMOS active pixel image sensor,” IEEE T. Electron. Dev. 41, 452–453 (1994). 30. A. Mehrish, A. Subramanyam, and S. Emmanuel, “Sensor pattern noise estimation using probabilistically estimated RAW values,” IEEE Signal Process. Lett. 23, 693–697 (2016). 31. K. Yonemoto and H. Sumi, “A CMOS image sensor with a simple fixed-pattern-noise-reduction technology and a hole accumulation diode,” IEEE J. Solid-St. Circ. 35, 2038–2043 (2000). 32. A. J. Cooper, “Improved photo response non-uniformity (PRNU) based source camera identification,” Forensic Sci. Int. 226, 132–141 (2013). 33. M. J. Schulz and L. V. Caldwell, “Nonuniformity correction and correctability of infrared focal plane arrays,” in “SPIE’s 1995 Symposium on OE/Aerospace Sensing and Dual Use Photonics,” (International Society for Optics and Photonics, 1995), pp. 200–211. 34. D. Litwiller, “CCD vs. CMOS,” Photon. Spectra 35, 154–158 (2001). 35. B. E. Stine, D. S. Boning, and J. E. Chung, “Analysis and decomposition of spatial variation in integrated circuit processes and devices,” IEEE T. Semicond. Manuf. 10, 24–41 (1997). 36. N. Ricquier and B. Dierickx,“Active pixel CMOS image sensor with on-chip non-uniformity correction,” in “Proc. IEEE Workshop Charge-Coupled Devices and Advanced Image Sensors,” (1995), pp. 20–22. 37. A. Piva, “An overview on image forensics,” ISRN Sig. Proc. 2013 (2013). 38. M. Sheng, J. Xie, and Z. Fu, “Calibration-based NUC method in real-time based on IRFPA,” Physics Procedia 22, 372–380 (2011). 39. D. L. Perry and E. L. Dereniak, “Linear theory of nonuniformity correction in infrared staring sensors,” Opt. Eng. 32, 1854–1859 (1993). 40. S. N. Torres, E. M. Vera, R. A. Reeves, and S. K. Sobarzo, “Adaptive scene-based nonuniformity correction method for infrared-focal plane arrays,” in “AeroSense 2003,” (International Society for Optics and Photonics, 2003), pp. 130–139. 41. C. Zuo, Q. Chen, G. Gu, and X. Sui, “Scene-based nonuniformity correction algorithm based on interframe registration,” JOSA A 28, 1164–1176 (2011). 42. S. Saha, “Image compression-from DCT to wavelets: a review,” Crossroads 6, 12–21 (2000). 43. Y.-M. Zhou, C. Zhang, and Z.-K. Zhang, “An efficient fractal image coding algorithm using unified feature and DCT,” Chaos, Solitons & Fractals 39, 1823–1830 (2009). 44. E. J. Candès and M. B. Wakin, “An introduction to compressive sampling,” IEEE Sig. Proc. Mag. 25, 21–30 (2008). 45. E. Candes and J. Romberg, “l1-magic: Recovery of sparse signals via convex programming,” URL: www. acm. caltech. edu/l1magic/downloads/l1magic. pdf 4, 14 (2005).